Skip to content

fix(service): Cap known values of runs without a token, key write-token runs on the main thread - #48

Merged
dcramer merged 3 commits into
mainfrom
fix/dos-limits-review-2
Nov 5, 2026
Merged

dcramer merged 3 commits into
mainfrom
fix/dos-limits-review-2

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #45. It fixes the blockers from the review of #45 (head 411bc6b). Keys do not change: the pin tests pass on b27f425 and on this head.

Changes

  • Per-run known cap for runs without a token. All sessions of a run count together: at most 10,000 values and 256 Ki characters. A request that would go past the cap fails with 400, and the run ends. A value is never dropped, so no key changes. The count of a session goes away when the session ends or is evicted. Runs with the token have no cap.
  • Runs with the token key on the main thread again, as on main. Their misses are also redacted there, with the key time limit. The key worker serves only runs without a token, in FIFO order. Its flags are --max-old-space-size=256 --max-semi-space-size=4.
  • Values over 32,767 characters stay out of known and out of templates. V8 cannot compile a pattern with such a literal, so before, every later request of the session failed with 502. On b27f425 those requests never got a key either, so to leave the values out changes no key.
  • extractValues() counts each match when it finds it (an exec()/lastIndex loop), so the cap of 100,000 values fails the request with 400 before memory grows. A test shows that it finds the same values as replace().
  • normalizeJson() writes the key text without a copy of the body. Thinking blocks are marked in place. A differential check of 20k random cases against b27f425 found 0 key differences.
  • clientGroup() maps IPv4-mapped IPv6 addresses in all forms to IPv4: ::ffff:a.b.c.d, ::ffff:c000:201, and 0:0:0:0:0:ffff:….
  • README:
    • The cap and the >32,767 rule.
    • The scope of the worker.
    • The IPv4-mapped grouping and the /48 gap.
    • A split deploy order for the PROXY header, so that no request fails. The order is: config secret with -target, then the VM reset, then the full apply.

Cap data

  • In 14 days, 59 runs without a token had at most 11 requests and 4,376 input tokens each. The largest model request of any run had 162,707 input tokens (about 0.65 MB).
  • In Junior prod model requests, the eval patterns found at most 768 values and 14.7 K characters per MiB. That is about 500 values and 10 K characters for each session. The caps are about 20× that.
  • 100 runs at the cap hold about 200 MiB.

Before (411bc6b) → after

Repro Before After
64 sessions × 15 MB of distinct [a-z]{150} main heap 1.27 GB, RSS 1.92 GB in total, the run never ends first request 400, the run ends, 386 MB in total
64 MiB of small text blocks, with the token 400, worker SIGABRT 200 in 8.9 s, RSS 1.14 GB (main: 10.3 s, 2.1 GB)
Value over 32,767 chars later requests 502, the run stays open 412 / 200
{v:"x"} with 8 MiB of x worker SIGABRT, 1.9 s 400 in 0.37 s, worker 134 MB
"x " × 4 Mi worker SIGABRT, 1.7 s 400 in 0.34 s, worker 114 MB

The worst job of a run without a token (16 MiB of {}) peaks at 714 MB in total, for the main process and the worker together. 10 runs at the cap plus that job: 734 MB.

Six new tests fail on 411bc6b and pass here.

RoachServiceConfig does not change, so deploy/gcp needs no change.

via @dcramer.

--

View Session [Sentry]

…en runs on the main thread

Address the review of #45:

- Count `known` across all sessions of a run without the token (10,000
  values, 256 Ki characters). A request past it fails with 400 and ends
  the run. No value is dropped, so no key changes.
- Key and redact runs with the token on the main thread, as on main. The
  worker serves runs without the token only, with a 256 MiB old space
  and a 4 MiB semi-space.
- Leave values over 32,767 characters out of `known` and templates. V8
  cannot run a pattern with them, so those requests failed before.
- Count matches in `extractValues()` as it finds them, so the cap of
  100,000 values stops before memory grows.
- Write the key text of a body without a copy of it.
- Group IPv4-mapped IPv6 addresses in all forms as IPv4.
- README: split deploy order for the PROXY header, the /48 gap.

Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
sentry-junior Bot and others added 2 commits November 5, 2026 07:45
Address the review of #48:

- Runs without the token share a budget of 128 MiB of request bodies.
  A request holds its bytes from when they come until it finishes, also
  while its key waits for the key worker. A request that does not fit
  gets 503 with `Retry-After: 1`, and the run goes on. A client that
  declares a large body and then stops holds only what it sent.
  30 and 60 parallel 16 MiB bodies now peak at about 1 GB of RSS for the
  service and the worker, down from 1.9 GB and 3.4 GB.
- README: the deploy order resets the VM twice, and a reset ends open
  runs. Document the body budget.
- Fix the doc comment of `canBeKnown()`: V8 fails on any text.

Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
…t slow bodies

One client that started a run without a token could hold all 128 MiB of
the body budget: 8 tunnels that each sent 16 MiB - 1 KiB and then one
byte now and then. Node does not time out requests in a tunnel, because
the tunnel server does not listen, so nothing gave the bytes back. Every
other run without a token then got HTTP 503.

- The runs of one client (`clientGroup()` of the client that started
  the run) can hold 32 MiB of the budget.
- A body of a run without a token that does not come in 30 s gets HTTP
  408, the connection closes, and the bytes go back.
- Count a 503 as `roach.request` with `roach.result:busy`, and a 408 as
  `roach.result:timeout`, and add a TELEMETRY.md query.
- README: the new limits, the memory bound for write bodies of a normal
  size, and the gap that Node does not time out tunneled requests.

Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
@dcramer
dcramer marked this pull request as ready for review November 5, 2026 15:33
@dcramer
dcramer merged commit 6c64d29 into main Nov 5, 2026
15 checks passed
@dcramer
dcramer deleted the fix/dos-limits-review-2 branch November 5, 2026 15:33
sentry-junior Bot added a commit that referenced this pull request Nov 5, 2026
Keep both sides of #48 and of this branch:
- server.ts: `readBody` clears the 30 s read deadline on end and on an
  error, and an ECONNRESET of a request still becomes the 400 `aborted`
  error. `answerError` counts `aborted`, `busy` (503), and `timeout`
  (408) as `roach.request`. The comment of the read deadline now says
  why it is shorter than Node's 300 s.
- service.test.ts: keep both sets of helpers. `startBody` uses
  `openTunnel`. The aborted test also checks that a closed request gives
  its bytes of the budget back.
- README.md: one metric row with all results. Remove the gap about Node
  time limits, because this branch fixes it.
- TELEMETRY.md: Busy, Timeout, and Aborted, and leave all three out of
  the hit rate.
dcramer added a commit that referenced this pull request Nov 5, 2026
CI already runs on push to `main` (`ci.yml` has `push: branches: [main,
"release/**"]`). But GitHub started no push workflows for `09f6582` (the
#47 merge): no CI, no Image, no CodeQL. The Deploy run of `9b2ca6a`
(#49) then saw a newer head and stopped ("main is at 09f6582 now, not
9b2ca6a. Nothing deploys."). So the head of `main` has no CI, no image
and no deploy, and `:main` still points at `6c64d29` (#48). Nothing
could start those runs again: neither workflow had `workflow_dispatch`.

**Changes**
- `ci.yml` and `image.yml`: add `workflow_dispatch`.
- `image.yml`: push `:<sha>` on every event except `pull_request`, so a
manual run on `main` makes the image.
- `deploy.yml`:
- `promote` accepts an Image run that isn't a `pull_request` (push or
manual). The `branches: [main]` filter still limits Deploy to `main`.
- "Wait for CI" moves to `.github/scripts/wait-for-ci.sh` and looks for
the CI run of the commit with `--branch main`, not `--event push`. A
manual CI run counts. A `pull_request` run of the same commit doesn't,
because its branch isn't `main`.
- `README.md`: "Deploy" says what to run when a push gets no workflows.

**After merge:** this merge is a new push, so it should get CI, Image
and Deploy normally. If it doesn't, run `gh workflow run ci.yml --ref
main` and `gh workflow run image.yml --ref main`.

**Checked**
- shellcheck (`--shell=sh --enable=check-extra-masked-returns`) and
actionlint pass. `oxfmt --check` passes.
- The script ran under busybox `ash` with stub `gh`, `timeout` and
`sleep`: success after two empty lists, a failed run, a run that never
ends (60 polls, then exit 1), a `gh` error, and no arguments (exit 2).
- The live query `gh run list --workflow ci.yml --branch main --commit
<sha>` finds the push run for `9b2ca6a`, nothing for `09f6582`, and
nothing for `3f11cb0` (#49's PR head, which has only a `pull_request`
run).

**Not changed:** the other inline shell in `deploy.yml` (head check, VM
reset, service wait). It can move to scripts in the service audit.

I can't confirm why GitHub missed the push. The commit message has no
skip marker, and every push workflow is missing, so it looks like a
GitHub delivery gap, not this repo's config.

<!-- junior-request-attribution:start -->
via @dcramer.
<!-- junior-request-attribution:end -->

<!-- junior-session-footer:start -->
<!-- junior-conversation-id:slack%3AC0ACFA5JBDX%3A1793830221.324719 -->

--

[View
Session](https://junior-prod.sentry.dev/conversations/slack%3AC0ACFA5JBDX%3A1793830221.324719)
[[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC0ACFA5JBDX%3A1793830221.324719/?project=4510944073809921)

<!-- junior-session-footer:end -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant